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ABSTRACT 


Pure autoregressive (AR) models which are linear in the short term, that is when a 
variable can be predicted by linear regression on a limited number of past observations, are 
discussed. When evenly-spaced observations are available, a fixed set of AR coefficients 
can be calculated independent of the data. For filtering purposes, such a lag structure can 
be implemented recursively with an efficient algorithm. The method of computing variance 
recursively is also derived. A complete algorithm is presented in the appendix. 
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INTRODU C TION 


Many important problems require the prediction of a variable sequentially based 
only on previous observations on that variable. The classic autoregressive (AR) model is 


n 

yk - X “tfk-i + w k ’ 

i-1 


0 ) 


Wk being a second moment ergodic white noise sequence (see for instance Groupe [1984] 
Chapter 8). Several models with a different cq are well known. Irving Fisher (1) studied 
a distributed lag structure with lag length n, of the following form: 



1 


n 



[nyk-1 + (n- 1 ) yk-2 + + yk-nl- 


( 2 ) 


This yields lag coefficients oti, 


tti = 



2(n+l-i) 
~ n(n+l) 


which insures that 



a\ =1 


(3) 


(4) 



The average lag (defined as the sum of the weighted time periods) is 


( 5 ) 


( 6 ) 

Another commonly used expectations model is called naive or static expectations, 
yk = yk-i> 

which is the Fisher equation with n=l. 

Other forms of expectations operators, which have been investigated principally 
with reference to price expectations, include extrapolative, adaptive, and various ad hoc 
distributed lags. Some empirical tests have been done: see Tumovsky (3) and Tumovsky 
and Wachter (4). 


n 


n 


X (ni+i-i 2 ) 

“=X ia i = i= H 

i =1 £ i 


i=l 


Then 


n 


I * 2 


a = n+1 - ~ — = n+1 - 

li 

i=l 


2n+l n+2 

T “ = T 
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An alternative method of forming expectations is by defining a linear trend using a 
specific number of observations, i.e.: 

y£ = a e + b e xk + ek (8) 

where a e and be are estimated from the data. 

By choosing the x scale so that xk=0 and the independent variables of the past 
observations are Xk-i = -i, the model becomes y^ = a e + b e • 0 = a e . We now show how to 

obtain the prediction for yk without actually estimating a orb, as follows. The least 
squares estimates of a and b are: 

, „ n I xi yi - L xi E y i 
n I x j - ( S xj )2 
and 

a e = y - b e x . 


(9) 

( 10 ) 


Letting D = nExj 2 - (Exj) 2 and substituting D into (9) and then into (10) yields 


Eyi n(Exj yj )Exj (Exj) 2 Eyj 
n nD + nD 
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Solving for The Lag Coefficients 


Next, reorder and expand terms in (11) to obtain: 

e 11 1 
a' = jyi + J 11 * + n yn 


(Eci ) 2 ffxi ) 2 (Zxj ) 2 

+ - iiI5 — 11 +T5 - » + + "135“ In 


Zx; Zx; 

(•D-^yi -<-2)- B i y 2 - 


Exi 

TT yn- 


( 12 ) 


Now we wish to obtain a e - aiyj + 0^2 + + any n - 

Summing the coefficients of yj above, we obtain the required AR coefficents which 
are, of course, independent of y: 


a j 


1 

n + 



n 

n|^*£(-i) 2 


+ 

n \2 

m 


n 


n*Z(-i) 2 

i=l 


■(i'J 


(13) 


Substituting the equations for sums of numbers and sums of squares into (13) 
yields a simplified expression for the AR coefficients: 




1 + 


3(n+l) 

(n-iy 



2(2n+l-3i) 

' n(ri- l) J 


(14) 


The important feature of this model is that it provides an AR lag structure whose 
coefficients follow direcdy from the hypothesis of the limited memory linear least squares 
model, and depend only on the order of the model and not on the data. Table 1 provides 
the coefficents from (14) for lag lengths up to 8. 


5 



TABLE 1. Linear Least Squares Expectations Lag Coefficients 
Age of Observation (in Time Periods) 



1 

2 

3 

4 

5 

6 

7 8 

2 

2 

-1 






3 

4/3 

1/3 

- 2/3 





4 

1 

1/2 

0 

- 1/2 




5 

8/10 

5/10 

2/10 

- 1/10 

- 4/10 



6 

2/3 

7/15 

4/15 

1/15 

- 2/15 

- 1/3 


7 

4/7 

3/7 

2/7 

1/7 

0 

- 1/7 

- 2/7 

8 

1/2 

11/28 

2/7 

5/28 

1/14 

- 1/28 

- 1/7 - 1/14 
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The average lag of the least squares expectations operator is 
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Predictions for Different Time Periods 


Coefficients generated by (14) provide least squares expectations for one 
observation after the last Coefficients can also be obtained to predict y further in the 
future. Let m be the number of observations missing between the last observation and the 
prediction period. If the index of the required expectation is zero, then 


b e = 


(n-m) Exj yj - Exj Exj 
(n-m) Exj 2 . ( £xj )2 


(16) 


where all summations are from m+1 to n. 
Then, let 


X 2 ( n 

x i ’ I*i 

i=m+l li=m+l 


= (n-m) 


f n m 

2 


Vi =1 


i N 



1 2 


n m \2 

( Xi 


E Xi ' E Xi 

1 J 


li=l »=1 J 


(17) 


Substituting for summations using the appropriate equations yields 
D = (n-m)2 (n-m-l)(n-m+l)/12. 


(18) 
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With some manipulation, the expression for the lag coefficients for expectations m 
observations forward is then 


1 (n(n+l) - m(m+l)}2 (n(n+l) - m(m+l)} (m+j) 

a J n-m 4(n-m)D 2D 


(19) 


For example, if expectations for three periods in the future are needed, and a lag 
length of four is chosen, then n=6 and m=2. Using (13) yields 

y£ = l- 6 yk-3 + 0.7y k -4 - 0*2y k . 5 -l.ly k . 6 . 


The sum of the lag weights is the same as when m=0, 


n-m 

X n . _ y ( 1 (n(n+l) -m(m+l)}2 (n(n+l) - m(m+l)} (m+i)^ 

=1 J ' i*m+l l” 1 ” Jd j 

J 


n-m (n-m){n(n+l) - m(m+l)}7 (n(n+l) - m(m+l)}2 

n-m + 4(n-m)D * 4D 


= 1 . 


( 20 ) 


The average lag of the future-period least squares expectations operator calculated 


X (m+ j) ctj 

j=l 

is also zero, but the proof is left to the reader. Table 2 shows coefficients for m=0 to 5 and 
order 2 to 5. 
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Table 2: Least Squares Expectations Coefficients Where m Periods Are 
Skipped Between the Last Observation and the Prediction. 


Number of Lag Coefficients (n-m) 


2 3 4 5 



2 

-1 

4/3 

1/3 

-2/3 

1 

.5 

0 

-.5 

.8 

.5 

.2 

-.1 

1 

3 

-2 

11/6 

1/3 

-7/6 

1.3 

.6 

-.1 

-.8 

1 

.6 

.2 

-.2 

2 

4 

-3 

7/3 

1/3 

-5/3 

1.6 

.7 

-.2 

-1.1 

1.2 

.7 

.2 

-.3 

3 

5 

-4 

17/6 

1/3 

-13/6 

1.9 

.8 

-.3 

-1.4 

1.4 

.8 

.2 

-.4 

4 

6 

-5 


1/3 

-8/3 

2.2 

.9 

-.4 

-1.7 

1.6 

.9 

.2 

-.5 
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Filtering, Applications 


In this section we show how the lag operator is implemented recursively, for the 
case m=0. 

£ 

First, find two successive estimates yk-l e and y^: 
n n-1 

y k -i = .1, «i Yk-l-i “ <*n yk-l- n + yk-l-i ( 21 ) 

i=l i=l 

and 

e n n-1 

y k = 2<*i yk-i = .^“i+l yk-l-i 

i=l i=0 


n-1 

= «1 yk -1 + .2, <*i+l yk-l-i • < 22 ) 

i=l 


The difference between (21) and (22) is 


yk ' yJc -1 = a l yk-1 ■ “n yk-l-n + 2 (Oi+l - «i ) yk-l-i- 


(23) 


But we note that 


«i+i - Cli 


= 2 


r-3(i+l) + 3i 

. n(n-l) 



(24) 


Substituting (24) into (23) we obtain the recursive expression 


yk = yk -1 + a l yk-1 • a n yk-l-n - 


6 n ‘^ 

H5TTT i 5i yk ' 1 ' i 


( 25 ) 


The recursion is completed by noting that 
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( 26 ) 


n-1 n-1 

.1 yk-l -i = I y k-2-i + yk-2 - yk-l-n • 

i=l i=l 


The recursive approach requires that the n most recent observations be stored, but at 
each iteration only y^.j, y k _ 2 , and yk-l-n enter die calculations. Except for startup 

processing (the first n observations) the amount of processing for such a filter is 
independent of the lag period n. An algorithm is provided in the appendix. 

Variance of the Prediction 


We now derive the variance of the prediction, a^y^. This variance is determined as 


follows (from Kmenta, p 228). 


V = E 


(Vk - E (y k)j*J 


= E {[(a e + b e x k ) - (a + bx k )] 2 } 


= E { (a e - a) 2 ) + E { (b e - b) 2 x£ } + 2E ( (a e - a)(b e - b) x k } 

= Var (a e ) + x^ Var(b e ) + 2x k Cov (a e , b e ) (27) 
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Now we know from linear regression that, where x’j = xj- x, 
o 2 


Var (b e ) =• 




Var (a e ) = a 2 


1 + x 2 "l 


and 


Cov(a e , b e ) = - x 


t2 1 


E Y 


where o 2 is Var (e^) in (8). 
Substituting, we obtain 


(28) 


_2 e 

° 2 yk : 


2 1 x 2 

2 

’ a 2 " 

‘ 2x k x 

£ 

" + 2x i 2 

+ x k 

Exj 2 

Exj 2 


a 2 y^ = a 2 | 


i + x 2 + * k 2x k x 

n *'t.2 Ex?2 y.v 2 


Ex^ Ex^ Ex^ 


a 2 yJ = o 2 l 


1 (Xfc - x) 2 


n + 


Zx i 


.2 


(29) 


The expression in (29) gives the variance of the predicted mean value of y for a 
given xk- Since the actual observed value of y varies about the true mean value with 

variance a 2 (independent of the variance of y^), the predicted value of an individual 

observation will still be given by y^ but will have variance (from Draper and Smith, p.24): 
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a 2 + G 2y® - <y2 


(30) 


and since x^ = 0, 


i + -+ <Xk 
n 


w 


(Xk- x) 2 = (- x )2-ls^l£ 

So, 


o 2 yk = a 2 


i + 

n 



or, 




(31) 


(32) 


(33) 


Whether (32) or (33) is the appropriate equation will depend on the application. If 
the filter is predicting the mean value of y (such as the actual position of a target) then (32) 
should be used, because the variance can be made arbitrarily small by increasing the 
number of observations. Conversely, if we need to predict the next observation such as for 
certain search applications, the (33) is used and the minimum variance is no matter how 
many observations are used. In the remaining analysis we will use (33), however the 
development using (32) is nearly identical. 
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Variance of the Disturbance 


i _ 

To estimate o 2 we use s 2 , an unbiased estimator where )| = yj - y: 


t n 

s 2 = - a ® • b ® x i) 2 • 

i=l 


n 


= «^S[ yi ‘ ( yi * be x)_bex (f 

t=i 


n 


= (y i ' beXi)2 

i=l 


n 


n 


ss - 2t > 2 "S' x i y\ + < be > 2 £ x i 2 

. i=1 Tf i=1 . 


But we know (see Kmenta, p. 208) that 


n n 


be 


^ 2 -I‘i y i 
1=1 


and 


n 


n 


(b e ) 2 ^ *i 2 = b e xj yj . 
i=1 i^l 
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So we obtain. 



( 34 ) 


which is a well-known expression for s^. The limited-memory filter is a unique model 
where the x\ are known, therefore, some simplification is possible. First, express 


E*i Yi 



and subsitute it into (34) to yield. 




(35) 


Now, it is easy to show that 
n 



which yields 
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Zyl-i - n 


/ n 


2>k-i 


:V - 


i=l 


Z' n 




1 


n 



(37) 


For an efficient recursive calculation of s 2 , we do not wish to calculate the 
summations in (37) each time. Of course, since n is constant and x^-i = -i, 


n 



n(n+l)(2n+l) 

5 


n(n+l) 2 

3 


(38) 


Furthermore, 
n n 


£ yk-i = J yk-l-i - yk-l-n + yk-1 

i^l i=l 


(39) 


and 



y k-l-i ' y k-l-n + y k-l * 


(40) 


Now, to compute the final summation term recursively, we write 
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( 41 ) 



Now, we expand the first tem in (41), as follows, for epochs k and k-1: 


n 

- X 1 y k-i = ^k-l - 2 Yk-2 ' - ’ n yk-n 
i=l 


n 

- Vi yk-l-i = _ yk-2 ' ••• * ( n_1 )yk-n - n yk-l-n- 
i=l 


Then we subtract, yielding 
n n n 

-Vi yk-i + y i yk-l-i = -y yk-i + n yk-l-n* 

i=l i=l i=l 


(42) 


Now we write the expression for the cross product term at epoch k and epoch k-l, 
using (41) 


f n 

t t 

x i yi 

i 

\ i=1 j 


i* 

s 


n n+1 n 

-Y iyk-i+ -yXyk-i 
i=l i=l 


and 
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n 


n+1 


n 


-Siyk-i-i + Tjyk-i-i 

i=l i=l 


Subtracting the last two equations and substituting (39) and (42), we obtain finally, 
/ n \ f n \ 




£ yk-i + nyk-l-n + Tf(yk-1 - yk-l-n) 


(43) 


which is the last required difference equation. 

The algorithm which implements this recursive model is described in the appendix. 

Statistical Justification for the Model 

Under what circumstance is such a linear model valid? We can think of many real 
world situations where processes are linear (or can be transformed to ones that are) over the 
short term but in the long term may be very non-linear. One with which we are familiar is a 
linear process whose slope is subject to randomly occurring jumps. Under this 
assumption, we have the true model, 

yj = a + bxi + ej, i > k-n (44) 

where k-n is the epoch of the most recent jump. Of course, n is the unknown but we can 
estimate its value. This estimate may give us two kinds of problems in the estimation of a 
and b. First, if we choose n too large, and we include data points which are not part of the 
true model, our regression is biased. However, until the next jump occurs, we are at least 
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consistent. On the other hand, if we choose n too small, then we are omitting usable data 
from the regression, which is inefficient. 

We suggest that it might be possible, given a particular application, to choose n so 
as to minimize the expected total (bias and inefficiency) error. 

Computational Results 

Table 3 shows a simulated data set and the results of using the model with order 
n=8. The true model is yi = 18+2j, i=l,...,18 and yi = 54-i, i=19,...,50. The observation 

error introduced in the simulation is normal with a=3. 

Figure 1 shows the behavior of the model for n=6, 8, 10, and 15, respectively. 
Note that the calculated variance (using equation (32)) increases for a time (about n 
observations) after the jump at observation 18, an indication that the filter in some sense 
"detects" the jump. 

We should note that much more powerful filters are available to deal with the type 
of data discussed herein, such as Kalman Filters with short- and long-term processes, but 
all such models require considerably more processing and are much more general. The 
filter presented here is narrowly-defined but extremely easy to compute. 
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Table 1 

Simulation Results 


Number Observ Predict 


1 

22.3367 


2 

24.0347 


3 

29.6256 


4 

28.5831 


5 

27.4105 


6 

34.6796 


7 

35.4258 


8 

36.9346 


9 

33.2763 

39.1537 

10 

36.8247 

38.1588 

11 

42.0522 

38.3724 

12 

40.1222 

41.8262 

13 

44.3455 

42.6113 

14 

47.5987 

43.8529 

15 

41.1860 

47.5027 

16 

55.5352 

46.8285 

17 

51.9135 

53.2506 

18 

50.4021 

54.5623 

19 

52.7145 

54.3888 

20 

52.8151 

55.7838 

21 

51.2435 

55.3974 

22 

45.9523 

54.3735 

23 

47.6973 

51.1269 

24 

49.4900 

46.6388 

25 

46.8479 

47.4743 

26 

45.5741 

46.3428 

27 

42.9773 

44.5775 

28 

40.2959 

42.8569 

29 

44.4757 

40.8748 

30 

39.7585 

41.7626 

31 

38.6393 

39.1273 

32 

39.6311 

37.2192 

33 

41.7540 

37.4178 

34 

37.8070 

38.7905 

35 

37.6862 

38.2805 

36 

41.9468 

38.6149 

37 

37.6959 

38.6889 

38 

30.6637 

38.9533 

39 

34.6891 

34.7433 

40 

34.5748 

33.1207 

41 

30.1492 

32.2185 

42 

31.6565 

30.4034 

43 

30.3123 

29.1756 

44 

25.7946 

27.8916 

45 

25.7283 

26.6604 

46 

23.4342 

25.6085 

47 

18.8451 

22.1803 

48 

27.1559 

18.6698 

49 

22.6779 

20.9714 

50 

25.5353 

20.2484 


Error Variance Sigma 


5.8773 

4.009 

2.434 

1.3341 

4.358 

2.088 

- 3.6797 

3.792 

1.947 

1.7040 

4.200 

2.049 

- 1.7343 

4.368 

2.090 

- 3.7458 

3.195 

1.787 

6.3166 

3.498 

1.870 

- 8.7067 

5.834 

2.415 

1.3371 

9.065 

3.011 

4.1602 

8.911 

2.985 

1.6743 

9.730 

3.119 

2.9687 

9.683 

3.112 

4.1539 

9.419 

3.069 

8.4212 

10.218 

3.197 

3.4296 

14.498 

3.808 

- 2.8512 

2.465 

1.570 

.6264 

2.598 

1.612 

.7688 

2.556 

1.599 

1.6002 

1.986 

1.409 

2.5610 

2.147 

1.465 

- 3.6009 

2.474 

1.573 

2.0041 

2.991 

1.730 

.4880 

2.134 

1.461 

- 2.4119 

1.872 

1.368 

- 4.3363 

1.888 

1.374 

.9835 

2.721 

1.649 

.5944 

2382 

1.543 

- 4.3318 

2383 

1.544 

.9929 

3.434 

1.853 

8.2896 

2.105 

1.451 

.0542 

6.192 

2.488 

- 1.4540 

5326 

2.308 

2.0692 

5.110 

2.260 

-12531 

5.333 

2.309 

- 1.1367 

4.945 

2.224 

2.0970 

4.680 

2.163 

.9321 

3.552 

1.885 

2.1743 

3.241 

1.800 

3.3352 

1.086 

1.042 

- 8.4860 

1.590 

1.261 

- 1.7065 

5.798 

2.408 

- 5.2869 

5.918 

2.433 
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Results of Simulated Data Using Limited-Memory Filter 
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APPENDIX 


SEQUENTIAL ALGORITHM FOR Y e 


BEGIN 


and 


Select n; then, C 0 = n (n-l) ~ 
2(n+l-3i) 

a i “ n(n-l) ’ ' 


Get n observations. 


6 

Compute y n+1 = X «iyn+l-i 
and 

n-1 

X y n -i • 

i=l 


Set k=n+l . 


REPEAT 

Observe y fc. 

INC (k,l) 


Compute 


n-1 

X yk-l-i 

1=1 


/n-1 \ 

X y k-2-i + yk-2 -yk-l-n • 

U =1 ) 


Compute 


e /n-1 

yt = yh +a iyk-i -a n yk-i-n - c o . 


END 
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H M 


RECURSIVE ALGORITHM FOR VAR (Y e ) 


BEGIN 

Compute Ci = Ex' 2 = Si lt*) ( 2n+1 > . ”("+*> 2 


and 


C 2 = 1 + 


1 . (n+1) 2 
n + 4*Ci 



(n+1) 2 

Ter- 


cet n observations. 


Compute 


n 

I y n +l-i 

i=l 


n 2 

iJi y n +i-i • 


- i n 

y=n yn+i-i • 

i=l 


Exy 



x n+l-i 



and 



1 

nT 


n 



2 

y n +i-i 



(^x’y’ n+ i ) 

d 



Set k = n+1 . 
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REPEAT 


Observe y k . 

INC (k,l) . 

n n 

Compute # X yk-i = X Yk-l-i +yk-l - Yk-l-n . 

i=l i=l 

^ n+ 1 

Ix’y’ k = Ex’y’ k-1 - X yk-i + «y k-l-n + “T" (yk-1 - y k-l-n ) » 

i=l 

i?! y k-i = ill y k-l-i + (y k ~ l )2 ' (y k - 1 ’ n )2 ’ 


and 



END 
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